{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(parallel-magics)=\n", "\n", "# Parallel Magics\n", "\n", "IPython has a few magics to make working with your engines\n", "a bit nicer in IPython, e.g. in a Jupyter notebook.\n", "\n", "These are mainly shortcuts to {meth}`.DirectView.execute`\n", "and {meth}`.AsyncResult.display_outputs` methods, respectively.\n", "\n", "As always, first we start a cluster (or connect to an existing one):" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 4 engines with \n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6cc56f1a384d4ed286658cdb559c3210", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4 [00:00\n", " Group the outputs in a particular way. Choices are:\n", " **type**: group outputs of all engines by type\n", " (stdout, stderr, displaypub, etc.). **engine**:\n", " display all output for each engine together.\n", " **order**: like type, but individual displaypub output\n", " from each engine is collated. For example, if multiple\n", " plots are generated by each engine, the first figure\n", " of each engine will be displayed, then the second of\n", " each, etc.\n", " -e group outputs by engine (same as group-outputs=engine)\n", " -r collate outputs in order (same as group-outputs=order)\n", " --local also execute the cell in the local namespace\n", "\u001b[0;31mFile:\u001b[0m ~/dev/ip/parallel/ipyparallel/client/magics.py" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%px?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Targets\n", "\n", "You can specify where you want the code to run with the `--targets` argument.\n", "This lets you quickly set up different combinations of engines as appropriate.\n", "`--targets accepts slice syntax, so you can specify start, stop, and strides,\n", "just like you would slicing a list:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] I am even\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] I am even\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%px --targets ::2\n", "print(\"I am even\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:1] I am number 1\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%px --targets 1\n", "print(\"I am number 1\")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] still 'all' by default\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] still 'all' by default\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] still 'all' by default\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] still 'all' by default\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%px\n", "print(\"still 'all' by default\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When setting up your working environment, you might want to do some imports and define some functions _everywhere_, both on engines and here in your local notebook kernel.\n", "That's what `--local` is for:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "%%px --local\n", "import os\n", "import socket\n", "\n", "import psutil\n", "\n", "pid = os.getpid()\n", "p = psutil.Process()\n", "\n", "\n", "def report_info():\n", " return {\n", " \"hostname\": socket.gethostname(),\n", " \"pid\": os.getpid(),\n", " \"memory\": p.memory_info().rss,\n", " \"cpu\": sum(p.cpu_times()),\n", " }" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'hostname': 'heavy-2.local',\n", " 'pid': 25688,\n", " 'memory': 82870272,\n", " 'cpu': 0.828767104}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "report_info()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\u001b[0;31mOut[0:9]: \u001b[0m\n", "{'hostname': 'heavy-2.local',\n", " 'pid': 25705,\n", " 'memory': 78331904,\n", " 'cpu': 0.500559408}" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "report_info()", "execute_result": { "data": { "text/plain": "{'hostname': 'heavy-2.local',\n 'pid': 25705,\n 'memory': 78331904,\n 'cpu': 0.500559408}" }, "execution_count": 9, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:58.531039Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:9]: \u001b[0m\n", "{'hostname': 'heavy-2.local',\n", " 'pid': 25706,\n", " 'memory': 78675968,\n", " 'cpu': 0.50026576}" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "report_info()", "execute_result": { "data": { "text/plain": "{'hostname': 'heavy-2.local',\n 'pid': 25706,\n 'memory': 78675968,\n 'cpu': 0.50026576}" }, "execution_count": 9, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:58.531290Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:9]: \u001b[0m\n", "{'hostname': 'heavy-2.local',\n", " 'pid': 25707,\n", " 'memory': 78020608,\n", " 'cpu': 0.501814376}" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "report_info()", "execute_result": { "data": { "text/plain": "{'hostname': 'heavy-2.local',\n 'pid': 25707,\n 'memory': 78020608,\n 'cpu': 0.501814376}" }, "execution_count": 9, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:58.531514Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:8]: \u001b[0m\n", "{'hostname': 'heavy-2.local',\n", " 'pid': 25711,\n", " 'memory': 78954496,\n", " 'cpu': 0.489563088}" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "report_info()", "execute_result": { "data": { "text/plain": "{'hostname': 'heavy-2.local',\n 'pid': 25711,\n 'memory': 78954496,\n 'cpu': 0.489563088}" }, "execution_count": 8, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:58.531813Z" }, "output_type": "display_data" } ], "source": [ "%px report_info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Blocking (or not)\n", "\n", "You don't have to wait for results for every execution.\n", "`%%px --noblock` allows cells to return immediately, without waiting for the result:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 3.11 ms, sys: 2.61 ms, total: 5.72 ms\n", "Wall time: 6.25 ms\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "%%px --noblock\n", "import time\n", "time.sleep(2)\n", "pid, time.time()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But you will notice that this didn't output the result of the last command.\n", "For this, we have `%pxresult`, which waits for and displays the output of the latest request:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\u001b[0;31mOut[0:10]: \u001b[0m(25705, 1730717941.915563)" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:01.928172Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "import time\ntime.sleep(2)\npid, time.time()\n", "execute_result": { "data": { "text/plain": "(25705, 1730717941.915563)" }, "execution_count": 10, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_36", "outputs": [], "received": "2024-11-04T10:59:01.937149Z", "started": "2024-11-04T10:58:59.910427Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:59.896547Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:10]: \u001b[0m(25706, 1730717941.915562)" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:01.931585Z", "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "import time\ntime.sleep(2)\npid, time.time()\n", "execute_result": { "data": { "text/plain": "(25706, 1730717941.915562)" }, "execution_count": 10, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_37", "outputs": [], "received": "2024-11-04T10:59:01.938928Z", "started": "2024-11-04T10:58:59.911340Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:59.899876Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:10]: \u001b[0m(25707, 1730717941.918869)" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:01.931680Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "import time\ntime.sleep(2)\npid, time.time()\n", "execute_result": { "data": { "text/plain": "(25707, 1730717941.918869)" }, "execution_count": 10, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_38", "outputs": [], "received": "2024-11-04T10:59:01.937824Z", "started": "2024-11-04T10:58:59.912839Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:59.900945Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:9]: \u001b[0m(25711, 1730717941.9218612)" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:01.934283Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "import time\ntime.sleep(2)\npid, time.time()\n", "execute_result": { "data": { "text/plain": "(25711, 1730717941.9218612)" }, "execution_count": 9, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_39", "outputs": [], "received": "2024-11-04T10:59:01.939921Z", "started": "2024-11-04T10:58:59.915319Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T10:58:59.901237Z" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 13.5 ms, sys: 5.04 ms, total: 18.5 ms\n", "Wall time: 754 ms\n" ] } ], "source": [ "%time %pxresult" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Configuring defaults with `%pxconfig`\n", "\n", "The `%pxconfig` magic lets you configure the _default_ behavior of the `%px` magics.\n", "`%pxconfig` takes the same arguments as `%%px`, but saves them as the default for future calls to `%px` or `%%px`.\n", "\n", "In a notebook, I would generally suggest using `%pxconfig` at most once to set up your environment,\n", "and use `%%px --options` for cells that have different behavior than the default.\n", "\n", "It is much easier to see what code does and there is less of the dreaded \"hidden notebook state.\"\n", "\n", "`%pxconfig` and other line magics are principally for line-based environments like terminal IPython,\n", "where cell magics are impractical.\n", "\n", "For example, setting the default targets with `%pxconfig`:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:1] just 1\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%pxconfig --targets 1\n", "%px print(\"just 1\")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:1] 1 and 2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] 1 and 2\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%pxconfig --targets 1,2\n", "%px print(\"1 and 2\")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] back to all\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] back to all\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] back to all\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] back to all\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%pxconfig --targets all\n", "%px print(\"back to all\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Remote parallel magics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember, an IPython engine is IPython, so you can do magics remotely and in parallel as well!" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] Variable Type Data/Info\n", "-----------------------------------\n", "p Process psutil.Process(pid=25705,<...>ing', started='11:58:29')\n", "pid int 25705\n", "psutil module ages/psutil/__init__.py'>\n", "report_info function \n", "socket module ib/python3.11/socket.py'>\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] Variable Type Data/Info\n", "-----------------------------------\n", "p Process psutil.Process(pid=25706,<...>ing', started='11:58:29')\n", "pid int 25706\n", "psutil module ages/psutil/__init__.py'>\n", "report_info function \n", "socket module ib/python3.11/socket.py'>\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] Variable Type Data/Info\n", "-----------------------------------\n", "p Process psutil.Process(pid=25707,<...>ing', started='11:58:29')\n", "pid int 25707\n", "psutil module ages/psutil/__init__.py'>\n", "report_info function \n", "socket module ib/python3.11/socket.py'>\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] Variable Type Data/Info\n", "-----------------------------------\n", "p Process psutil.Process(pid=25711,<...>ing', started='11:58:29')\n", "pid int 25711\n", "psutil module ages/psutil/__init__.py'>\n", "report_info function \n", "socket module ib/python3.11/socket.py'>\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%px %whos" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Output\n", "\n", "IPython Parallel understands the Jupyter display protocol.\n", "When setting up a cluster, it is common for engines to want to know their own id or 'rank' in MPI parlance).\n", "\n", "You can do this with `scatter` and `flatten=True`:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "dv = rc[:]\n", "dv.scatter('rank', dv.targets, flatten=True)\n", "dv['stride'] = len(dv)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] 0 4\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] 1 4\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] 2 4\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] 3 4\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%px print(rank, stride)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] 0\n", "4\n", "8\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] 2\n", "6\n", "10\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] 1\n", "5\n", "9\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] 3\n", "7\n", "11\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:16]: \u001b[0mText(0.5, 1.0, 'Plot 1')" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "%config InlineBackend.figure_format = 'svg'\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nx = np.linspace(0, np.pi, 1000)\nfor n in range(rank, 12, stride): # noqa\n print(n)\n plt.plot(x, np.sin(n * x))\nplt.title(f\"Plot {rank}\") # noqa\n", "execute_result": { "data": { "text/plain": "Text(0.5, 1.0, 'Plot 1')" }, "execution_count": 16, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "1\n5\n9\n", "submitted": "2024-11-04T10:59:10.338184Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:13]: \u001b[0mText(0.5, 1.0, 'Plot 3')" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "%config InlineBackend.figure_format = 'svg'\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nx = np.linspace(0, np.pi, 1000)\nfor n in range(rank, 12, stride): # noqa\n print(n)\n plt.plot(x, np.sin(n * x))\nplt.title(f\"Plot {rank}\") # noqa\n", "execute_result": { "data": { "text/plain": "Text(0.5, 1.0, 'Plot 3')" }, "execution_count": 13, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "3\n7\n11\n", "submitted": "2024-11-04T10:59:10.338552Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:15]: \u001b[0mText(0.5, 1.0, 'Plot 2')" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "%config InlineBackend.figure_format = 'svg'\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nx = np.linspace(0, np.pi, 1000)\nfor n in range(rank, 12, stride): # noqa\n print(n)\n plt.plot(x, np.sin(n * x))\nplt.title(f\"Plot {rank}\") # noqa\n", "execute_result": { "data": { "text/plain": "Text(0.5, 1.0, 'Plot 2')" }, "execution_count": 15, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "2\n6\n10\n", "submitted": "2024-11-04T10:59:10.338378Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:14]: \u001b[0mText(0.5, 1.0, 'Plot 0')" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "%config InlineBackend.figure_format = 'svg'\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nx = np.linspace(0, np.pi, 1000)\nfor n in range(rank, 12, stride): # noqa\n print(n)\n plt.plot(x, np.sin(n * x))\nplt.title(f\"Plot {rank}\") # noqa\n", "execute_result": { "data": { "text/plain": "Text(0.5, 1.0, 'Plot 0')" }, "execution_count": 14, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "0\n4\n8\n", "submitted": "2024-11-04T10:59:10.337693Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2024-11-04T11:59:10.710350\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2024-11-04T11:59:10.710398\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2024-11-04T11:59:10.710387\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2024-11-04T11:59:10.710434\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" } ], "source": [ "%%px\n", "%config InlineBackend.figure_format = 'svg'\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "x = np.linspace(0, np.pi, 1000)\n", "for n in range(rank, 12, stride): # noqa\n", " print(n)\n", " plt.plot(x, np.sin(n * x))\n", "plt.title(f\"Plot {rank}\") # noqa" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%px --noblock\n", "import os\n", "import sys\n", "\n", "from IPython.display import HTML, Math, display\n", "\n", "\n", "def generate_output():\n", " \"\"\"function for testing output\n", "\n", " publishes two outputs of each type, and returns something\n", " \"\"\"\n", "\n", " print(\"stdout\")\n", " print(\"stderr\", file=sys.stderr)\n", "\n", " display(HTML(\"HTML\"))\n", "\n", " print(\"stdout2\")\n", " print(\"stderr2\", file=sys.stderr)\n", "\n", " display(Math(r\"\\alpha=\\beta\"))\n", "\n", " return os.getpid()\n", "\n", "\n", "generate_output()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When we retrieve the result, we can see the outputs" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[stdout:0] \n", "stdout\n", "stdout2\n", "[stdout:1] \n", "stdout\n", "stdout2\n", "[stdout:2] \n", "stdout\n", "stdout2\n", "[stdout:3] \n", "stdout\n", "stdout2\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[stderr:0] \n", "stderr\n", "stderr2\n", "[stderr:1] \n", "stderr\n", "stderr2\n", "[stderr:2] \n", "stderr\n", "stderr2\n", "[stderr:3] \n", "stderr\n", "stderr2\n" ] }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:15]: \u001b[0m25705" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:11.014407Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "import os\nimport sys\n\nfrom IPython.display import HTML, Math, display\n\n\ndef generate_output():\n \"\"\"function for testing output\n\n publishes two outputs of each type, and returns something\n \"\"\"\n\n print(\"stdout\")\n print(\"stderr\", file=sys.stderr)\n\n display(HTML(\"HTML\"))\n\n print(\"stdout2\")\n print(\"stderr2\", file=sys.stderr)\n\n display(Math(r\"\\alpha=\\beta\"))\n\n return os.getpid()\n\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 15, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_67", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:11.016530Z", "started": "2024-11-04T10:59:11.001550Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:10.988225Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:17]: \u001b[0m25706" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:11.015371Z", "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "import os\nimport sys\n\nfrom IPython.display import HTML, Math, display\n\n\ndef generate_output():\n \"\"\"function for testing output\n\n publishes two outputs of each type, and returns something\n \"\"\"\n\n print(\"stdout\")\n print(\"stderr\", file=sys.stderr)\n\n display(HTML(\"HTML\"))\n\n print(\"stdout2\")\n print(\"stderr2\", file=sys.stderr)\n\n display(Math(r\"\\alpha=\\beta\"))\n\n return os.getpid()\n\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25706" }, "execution_count": 17, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_68", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:11.017455Z", "started": "2024-11-04T10:59:11.003764Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:10.988672Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:16]: \u001b[0m25707" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:11.014525Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "import os\nimport sys\n\nfrom IPython.display import HTML, Math, display\n\n\ndef generate_output():\n \"\"\"function for testing output\n\n publishes two outputs of each type, and returns something\n \"\"\"\n\n print(\"stdout\")\n print(\"stderr\", file=sys.stderr)\n\n display(HTML(\"HTML\"))\n\n print(\"stdout2\")\n print(\"stderr2\", file=sys.stderr)\n\n display(Math(r\"\\alpha=\\beta\"))\n\n return os.getpid()\n\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25707" }, "execution_count": 16, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_69", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:11.017012Z", "started": "2024-11-04T10:59:11.008010Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:10.988885Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:14]: \u001b[0m25711" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:11.016214Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "import os\nimport sys\n\nfrom IPython.display import HTML, Math, display\n\n\ndef generate_output():\n \"\"\"function for testing output\n\n publishes two outputs of each type, and returns something\n \"\"\"\n\n print(\"stdout\")\n print(\"stderr\", file=sys.stderr)\n\n display(HTML(\"HTML\"))\n\n print(\"stdout2\")\n print(\"stderr2\", file=sys.stderr)\n\n display(Math(r\"\\alpha=\\beta\"))\n\n return os.getpid()\n\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25711" }, "execution_count": 14, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_70", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:11.017873Z", "started": "2024-11-04T10:59:11.009096Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:10.989072Z" }, "output_type": "display_data" } ], "source": [ "%pxresult" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%%px` also lets you choose some amount of the grouping of the outputs with `--group-outputs`:\n", "\n", "The choices are:\n", "\n", "* `type` - where stdout of each engine is displayed, then stderr, then rich output, etc. (the default)\n", "* `engine` - all of an engine's output is collected together\n", "* `order` - same as `type`, but individual displaypub outputs are interleaved.\n", " That is, it will output the first plot from each engine, then the second from each,\n", " etc.\n", "\n", "```{seealso}\n", "{meth}`.AsyncResult.display_outputs` for the grouping options.\n", "```\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:0] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:2] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:1] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:3] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:16]: \u001b[0m25705" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.222476Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 16, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_71", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.225924Z", "started": "2024-11-04T10:59:33.214730Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.205767Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:17]: \u001b[0m25707" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.222977Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25707" }, "execution_count": 17, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_73", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.226441Z", "started": "2024-11-04T10:59:33.216259Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206444Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:18]: \u001b[0m25706" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.223382Z", "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25706" }, "execution_count": 18, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_72", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.227627Z", "started": "2024-11-04T10:59:33.216035Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206258Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:15]: \u001b[0m25711" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.224117Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25711" }, "execution_count": 15, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_74", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.228377Z", "started": "2024-11-04T10:59:33.219181Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206615Z" }, "output_type": "display_data" } ], "source": [ "%%px --group-outputs=engine\n", "\n", "generate_output()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.\n", "\n", "\n", "`%pxresult` calls {meth}`.AsyncResult.display_outputs` on the most recent request.\n", "It accepts the same output-grouping arguments as %%px, so you can use it to view\n", "a result in different ways:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[stdout:0] \n", "stdout\n", "stdout2\n", "[stdout:1] \n", "stdout\n", "stdout2\n", "[stdout:2] \n", "stdout\n", "stdout2\n", "[stdout:3] \n", "stdout\n", "stdout2\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[stderr:0] \n", "stderr\n", "stderr2\n", "[stderr:1] \n", "stderr\n", "stderr2\n", "[stderr:2] \n", "stderr\n", "stderr2\n", "[stderr:3] \n", "stderr\n", "stderr2\n" ] }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:16]: \u001b[0m25705" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.222476Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 16, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_71", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.225924Z", "started": "2024-11-04T10:59:33.214730Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.205767Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:18]: \u001b[0m25706" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.223382Z", "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25706" }, "execution_count": 18, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_72", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.227627Z", "started": "2024-11-04T10:59:33.216035Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206258Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:17]: \u001b[0m25707" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.222977Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25707" }, "execution_count": 17, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_73", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.226441Z", "started": "2024-11-04T10:59:33.216259Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206444Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:15]: \u001b[0m25711" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:33.224117Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "\ngenerate_output()\n", "execute_result": { "data": { "text/plain": "25711" }, "execution_count": 15, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_74", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:33.228377Z", "started": "2024-11-04T10:59:33.219181Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:33.206615Z" }, "output_type": "display_data" } ], "source": [ "%pxresult --group-outputs=order" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Magics for multiple views\n", "\n", "All of the parallel magics are associated with a particular {class}`~.DirectView` object.\n", "You can change the active view by calling the {meth}`~.DirectView.activate` method\n", "on any view." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "even = rc[::2]\n", "even.activate()\n", "%px print(\"hi\")" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:2] hi\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:0] hi\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "even.block = True\n", "%px print(\"hi\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When activating a View, you can also specify a _suffix_, so that a whole different\n", "set of magics are associated with that view, without replacing the existing ones.\n", "This way, you can also have more than one set of parallel magics registered at a time.\n", "\n", "Let's restore the default magics DirectView and add activate a new set of magics one for a single engine." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "rc.activate() # restore default for %px\n", "e0 = rc[0]\n", "e0.block = True\n", "e0.activate(suffix='0')" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:0] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:19]: \u001b[0m25705" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:52.046818Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "generate_output()", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 19, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_79", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:52.048898Z", "started": "2024-11-04T10:59:52.035693Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:52.026609Z" }, "output_type": "display_data" } ], "source": [ "%px0 generate_output()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The default `%px` is still on all engines:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[stdout:0] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:0] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:2] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] stdout\n", "stdout2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:1] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stderr:3] stderr\n", "stderr2\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "HTML" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:2]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 2 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:0]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 0 }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:1]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 1 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:20]: \u001b[0m25707" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:58.874298Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "generate_output()", "execute_result": { "data": { "text/plain": "25707" }, "execution_count": 20, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_82", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:58.878013Z", "started": "2024-11-04T10:59:58.866480Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:58.853554Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[0:20]: \u001b[0m25705" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:58.874794Z", "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "generate_output()", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 20, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_80", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:58.878713Z", "started": "2024-11-04T10:59:58.863897Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:58.852885Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:19]: \u001b[0m25706" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:58.875032Z", "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "generate_output()", "execute_result": { "data": { "text/plain": "25706" }, "execution_count": 19, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_81", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:58.879919Z", "started": "2024-11-04T10:59:58.866506Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:58.853329Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "[output:3]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/latex": [ "$\\displaystyle \\alpha=\\beta$" ], "text/plain": [ "" ] }, "metadata": { "engine": 3 }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:16]: \u001b[0m25711" ] }, "metadata": { "after": [], "completed": "2024-11-04T10:59:58.876632Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "generate_output()", "execute_result": { "data": { "text/plain": "25711" }, "execution_count": 16, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_83", "outputs": [ { "data": { "text/html": "HTML", "text/plain": "" }, "metadata": {}, "transient": {} }, { "data": { "text/latex": "$\\displaystyle \\alpha=\\beta$", "text/plain": "" }, "metadata": {}, "transient": {} } ], "received": "2024-11-04T10:59:58.880366Z", "started": "2024-11-04T10:59:58.870708Z", "status": "ok", "stderr": "stderr\nstderr2\n", "stdout": "stdout\nstdout2\n", "submitted": "2024-11-04T10:59:58.853742Z" }, "output_type": "display_data" } ], "source": [ "%px generate_output()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For convenience, the {class}`~.Client` has a {meth}`~.Client.activate` method as well,\n", "which creates a DirectView with `targets='all', block=True`, activates it, and returns the new View.\n", "\n", "The initial magics registered when you create a Client for the first time are the result of a call to\n", "{meth}`rc.activate` with no arguments." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parallel Exceptions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you raise exceptions with the parallel exception,\n", "the CompositeError raised locally will display your remote traceback." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "tags": [ "raises-exception" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[1:execute]\n", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)\n", "Cell \u001b[0;32mIn[20], line 3\u001b[0m\n", "\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrandom\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m random\n", "\u001b[0;32m----> 3\u001b[0m A \u001b[38;5;241m=\u001b[39m \u001b[43mrandom\u001b[49m\u001b[43m(\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43minvalid shape\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:450\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:441\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random_sample\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32m_common.pyx:312\u001b[0m, in \u001b[0;36mnumpy.random._common.double_fill\u001b[0;34m()\u001b[0m\n", "\n", "\u001b[0;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer\n", "[3:execute]\n", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)\n", "Cell \u001b[0;32mIn[17], line 3\u001b[0m\n", "\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrandom\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m random\n", "\u001b[0;32m----> 3\u001b[0m A \u001b[38;5;241m=\u001b[39m \u001b[43mrandom\u001b[49m\u001b[43m(\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43minvalid shape\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:450\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:441\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random_sample\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32m_common.pyx:312\u001b[0m, in \u001b[0;36mnumpy.random._common.double_fill\u001b[0;34m()\u001b[0m\n", "\n", "\u001b[0;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer\n", "[0:execute]\n", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)\n", "Cell \u001b[0;32mIn[21], line 3\u001b[0m\n", "\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrandom\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m random\n", "\u001b[0;32m----> 3\u001b[0m A \u001b[38;5;241m=\u001b[39m \u001b[43mrandom\u001b[49m\u001b[43m(\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43minvalid shape\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:450\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:441\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random_sample\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32m_common.pyx:312\u001b[0m, in \u001b[0;36mnumpy.random._common.double_fill\u001b[0;34m()\u001b[0m\n", "\n", "\u001b[0;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer\n", "[2:execute]\n", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)\n", "Cell \u001b[0;32mIn[21], line 3\u001b[0m\n", "\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mrandom\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m random\n", "\u001b[0;32m----> 3\u001b[0m A \u001b[38;5;241m=\u001b[39m \u001b[43mrandom\u001b[49m\u001b[43m(\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43minvalid shape\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:450\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32mnumpy/random/mtrand.pyx:441\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.random_sample\u001b[0;34m()\u001b[0m\n", "\n", "File \u001b[0;32m_common.pyx:312\u001b[0m, in \u001b[0;36mnumpy.random._common.double_fill\u001b[0;34m()\u001b[0m\n", "\n", "\u001b[0;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer\n" ] }, { "ename": "AlreadyDisplayedError", "evalue": "4 errors", "output_type": "error", "traceback": [ "4 errors" ] } ], "source": [ "%%px\n", "from numpy.random import random\n", "\n", "A = random((100, 100, 'invalid shape'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, an error will occur on just one engine,\n", "while the rest are still working.\n", "\n", "When this happens, you will see errors immediately,\n", "and can interrupt the execution:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dv.scatter(\"rank\", rc.ids, flatten=True)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[0:execute]\n", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n", "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)\n", "Cell \u001b[0;32mIn[22], line 4\u001b[0m\n", "\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtime\u001b[39;00m\n", "\u001b[1;32m 3\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m rank \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m: \u001b[38;5;66;03m# noqa\u001b[39;00m\n", "\u001b[0;32m----> 4\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mrank 0 failed!\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", "\u001b[1;32m 5\u001b[0m time\u001b[38;5;241m.\u001b[39msleep(\u001b[38;5;241m10\u001b[39m)\n", "\n", "\u001b[0;31mRuntimeError\u001b[0m: rank 0 failed!\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "87b7c9c1dbaa492f8b16883f0b86edea", "version_major": 2, "version_minor": 0 }, "text/plain": [ "%px: 0%| | 0/4 [00:00 jupyter --existing \n", "or, if you are local, you can connect with just:\n", " $> jupyter --existing /Users/minrk/.ipython/profile_default/security/kernel-25705.json\n", "or even just:\n", " $> jupyter --existing\n", "if this is the most recent Jupyter kernel you have started.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:1] {\n", " \"shell_port\": 52987,\n", " \"iopub_port\": 52989,\n", " \"stdin_port\": 52992,\n", " \"control_port\": 52988,\n", " \"hb_port\": 52994,\n", " \"ip\": \"127.0.0.1\",\n", " \"key\": \"bc4c184f-ebc3b4399b253e678e8eb3e3\",\n", " \"transport\": \"tcp\",\n", " \"signature_scheme\": \"hmac-sha256\",\n", " \"kernel_name\": \"\"\n", "}\n", "\n", "Paste the above JSON into a file, and connect with:\n", " $> jupyter --existing \n", "or, if you are local, you can connect with just:\n", " $> jupyter --existing /Users/minrk/.ipython/profile_default/security/kernel-25706.json\n", "or even just:\n", " $> jupyter --existing\n", "if this is the most recent Jupyter kernel you have started.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:2] {\n", " \"shell_port\": 52990,\n", " \"iopub_port\": 52996,\n", " \"stdin_port\": 52998,\n", " \"control_port\": 52993,\n", " \"hb_port\": 52999,\n", " \"ip\": \"127.0.0.1\",\n", " \"key\": \"bc4c184f-ebc3b4399b253e678e8eb3e3\",\n", " \"transport\": \"tcp\",\n", " \"signature_scheme\": \"hmac-sha256\",\n", " \"kernel_name\": \"\"\n", "}\n", "\n", "Paste the above JSON into a file, and connect with:\n", " $> jupyter --existing \n", "or, if you are local, you can connect with just:\n", " $> jupyter --existing /Users/minrk/.ipython/profile_default/security/kernel-25707.json\n", "or even just:\n", " $> jupyter --existing\n", "if this is the most recent Jupyter kernel you have started.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[stdout:3] {\n", " \"shell_port\": 53002,\n", " \"iopub_port\": 53004,\n", " \"stdin_port\": 53005,\n", " \"control_port\": 53003,\n", " \"hb_port\": 53006,\n", " \"ip\": \"127.0.0.1\",\n", " \"key\": \"bc4c184f-ebc3b4399b253e678e8eb3e3\",\n", " \"transport\": \"tcp\",\n", " \"signature_scheme\": \"hmac-sha256\",\n", " \"kernel_name\": \"\"\n", "}\n", "\n", "Paste the above JSON into a file, and connect with:\n", " $> jupyter --existing \n", "or, if you are local, you can connect with just:\n", " $> jupyter --existing /Users/minrk/.ipython/profile_default/security/kernel-25711.json\n", "or even just:\n", " $> jupyter --existing\n", "if this is the most recent Jupyter kernel you have started.\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%px %connect_info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Legacy magics: `%autopx`\n", "\n", "Because IPython used to be a terminal-only, there are some leftover magics that are only meant for use in terminal IPython, where inputs are necessarily linear and the cost of re-typing things is higher.\n", "\n", "The use of `%autopx` is notebooks is __strongly__ discouraged.\n", "Even other stateful magics like `%pxconfig` should probaly not be used more than once in a notebook to set up defaults.\n", "Use `%%px` with arguments to make parallel cells clear and consistent and self-contained.\n", "\n", "The cost of `%autopx` is high (unclear notebooks, very surprising behavior on re-execution),\n", "and the benefit is small (saves a little typing)." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "%autopx enabled\n" ] } ], "source": [ "%autopx" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\u001b[0;31mOut[0:27]: \u001b[0m25705" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 0, "engine_uuid": "3b5836d1-d16d81c32917ce2147c20af7", "error": null, "execute_input": "os.getpid()", "execute_result": { "data": { "text/plain": "25705" }, "execution_count": 27, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T11:00:52.095079Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[1:26]: \u001b[0m25706" ] }, "metadata": { "after": null, "completed": null, "data": {}, "engine_id": 1, "engine_uuid": "3bda9160-2b8556074eb4c377b62e6f81", "error": null, "execute_input": "os.getpid()", "execute_result": { "data": { "text/plain": "25706" }, "execution_count": 26, "metadata": {} }, "follow": null, "msg_id": null, "outputs": [], "received": null, "started": null, "status": null, "stderr": "", "stdout": "", "submitted": "2024-11-04T11:00:52.095184Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[2:27]: \u001b[0m25707" ] }, "metadata": { "after": [], "completed": "2024-11-04T11:00:52.102695Z", "data": {}, "engine_id": 2, "engine_uuid": "db223688-6994b7bdbc25d90bddac66e7", "error": null, "execute_input": "os.getpid()", "execute_result": { "data": { "text/plain": "25707" }, "execution_count": 27, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_114", "outputs": [], "received": "2024-11-04T11:00:52.106305Z", "started": "2024-11-04T11:00:52.099286Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T11:00:52.095216Z" }, "output_type": "display_data" }, { "data": { "text/plain": [ "\u001b[0;31mOut[3:23]: \u001b[0m25711" ] }, "metadata": { "after": [], "completed": "2024-11-04T11:00:52.103031Z", "data": {}, "engine_id": 3, "engine_uuid": "d8b18943-9f945ec05692d14947250666", "error": null, "execute_input": "os.getpid()", "execute_result": { "data": { "text/plain": "25711" }, "execution_count": 23, "metadata": {} }, "follow": [], "is_broadcast": false, "is_coalescing": false, "msg_id": "dc435629-8bb652d0d09a5733938c7249_25688_115", "outputs": [], "received": "2024-11-04T11:00:52.107142Z", "started": "2024-11-04T11:00:52.099541Z", "status": "ok", "stderr": "", "stdout": "", "submitted": "2024-11-04T11:00:52.095443Z" }, "output_type": "display_data" } ], "source": [ "os.getpid()" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "%autopx disabled\n" ] } ], "source": [ "%autopx" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Linting and formatting notebooks with IPython Parallel magics\n", "\n", "Increasingly, Python formatting and linting tools like `black` and `ruff` aim to support Jupyter notebooks with IPython kernels.\n", "These tools work best if the notebooks don't use any IPython syntax extensions like magics,\n", "but they have varying degrees of support for understanding magics that may contain Python code.\n", "\n", "`black` can be taught that `%%px` contains Python code to format with the [python-cell-magics option]\n", "\n", "```toml\n", "[tool.black]\n", "python_cell_magics = [\n", " \"px\",\n", "]\n", "```\n", "\n", "`ruff` does not recognize `%%px` and its cell magic support is not configurable (as of 0.7.1),\n", "but may [gain a similar feature soon](https://github.com/astral-sh/ruff/issues/13977).\n", "\n", "`nbqa` can be used to invoke ruff, and _can_ be taught about cell magics like black:\n", "\n", "```toml\n", "[tool.nbqa.process_cells]\n", "# formatting is usually a good idea\n", "\"ruff format\" = [\n", " \"px\",\n", "]\n", "# check may not be\n", "\"ruff check\" = [\n", " \"px\",\n", "]\n", "```\n", "\n", "Running formatters on `%%px` is generally safe, but running linters that try to identify namespace issues is always going to be a challenge because, by their nature, parallel magics are working in multiple namespaces and the linters aren't going to understand that.\n", "\n", "Issues likely to come up:\n", "\n", "In general:\n", "\n", "If your linter _doesn't_ process `%%px` cells:\n", "\n", "- If you use `%%px --local` to define functions or variables, and your linters _don't_ process the `%%px` cells, then you are likely to face undefined name errors when you use the variables in local cells (e.g. F821)\n", "\n", "If your linter _does_ process `%%px` cells:\n", "\n", "- If you use `%%px` _without_ `--local`, the linter may erroneously identify your remote imports as duplicates (F401) to _reference_ things defined in single-cell. It's usually not too bad to put a `# noqa` on imports for this\n", "- If you define variables with `push` or `%px` and then reference them with `%%px`, you will likely see incorrect unused variable errors (F821).\n", " This is harder to work around with `# noqa` since there are likely multiple references.\n", " You can use a no-op assignment of a variable to itself, like `a = globals()['a']` which should convince the linter the variable is defined, but is annoying.\n", "\n", "The most straightforward way to address these is to run formatters but not linters on `%%px` cells, and do not use `%%px --local`.\n", "If you do want to use `%%px --local`, it will probably work best if you _do_ enable linting on `%%px`, but be careful in how you arrange your imports.\n", "\n", "You can also use per-file-ignores to disable namespace-checking lint (F8, F4).\n", "\n", "The example notebooks in the IPython Parallel repository currently run `ruff` via nbqa, and do enable linting,\n", "and you can check out the nbqa and ruff configuration used." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "01ce2f8f2b044624a7935dde29b807e4": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "02b2fb2533f843138f934a50570922b1": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "0c41ca80d0834a129f9ee68f5e6f46c5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_958c7ee6595f4dfbbab8d4366d4fbfd5", "max": 4, "style": "IPY_MODEL_f889c274f45f40f39d285ffdf81d8927", "value": 4 } }, "18d2f36a79cb464493c4bb210c7f2d3d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_f3f1103dc3f34671833adbe0b2dc07f4", "style": "IPY_MODEL_8ff495a1bd45497dbd7cdbd2c36e2080", "value": " 4/4 [00:01<00:00,  3.07tasks/s]" } }, "2397c7cfcd6b477d990d8e87ebee740b": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "2916a04172c14908a8fca98f367583eb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "2a1aa906910742e88b0e8dc8cc865c3c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_d4df9e677d6d47f88dcc88680753f87f", "style": "IPY_MODEL_2916a04172c14908a8fca98f367583eb", "value": " 4/4 [00:00<00:00,  4.03engine/s]" } }, "38cf14d49e554edaa17109a726ff70b5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_b851d6883c9f466ebe94d70bf9f9f8d6", "style": "IPY_MODEL_7524d5a02dd84ac289e685908fb99adb", "value": " 4/4 [00:08<00:00,  2.16s/tasks]" } }, "3e6a83f852b34d099ebee7ce2b994e80": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_f899c73fab574bde9f0e06c68d2c6f65", "style": "IPY_MODEL_5fba2ecdd37d4aedb0c39c1dd05332ca", "value": "%px: 100%" } }, "525f9918fc0e4dc2a955a475c6372083": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "5e30fa0143c9476f89f33bc05a27e4f7": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "5fba2ecdd37d4aedb0c39c1dd05332ca": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "6746fd5defb744b7a2b942e5ac957881": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_02b2fb2533f843138f934a50570922b1", "style": "IPY_MODEL_801c64f43c96412dbef731a87f404969", "value": "%px: 100%" } }, "6b9440ff0889439fadfd6ac2eed48e6b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_2397c7cfcd6b477d990d8e87ebee740b", "max": 4, "style": "IPY_MODEL_db29d3f772b54102932d4264cbed0f67", "value": 4 } }, "6cc56f1a384d4ed286658cdb559c3210": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_7dc7b51edbb9468ea980a6a330c9845a", "IPY_MODEL_6b9440ff0889439fadfd6ac2eed48e6b", "IPY_MODEL_2a1aa906910742e88b0e8dc8cc865c3c" ], "layout": "IPY_MODEL_5e30fa0143c9476f89f33bc05a27e4f7" } }, "6fb9232163254f16a9b3ca6c9709e373": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "7524d5a02dd84ac289e685908fb99adb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "7dc7b51edbb9468ea980a6a330c9845a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_525f9918fc0e4dc2a955a475c6372083", "style": "IPY_MODEL_6fb9232163254f16a9b3ca6c9709e373", "value": "100%" } }, "801c64f43c96412dbef731a87f404969": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "86c7d46257bf44d3a8bf5084e8dc142a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_01ce2f8f2b044624a7935dde29b807e4", "max": 4, "style": "IPY_MODEL_dd77821774f64a42addbd009fc56e952", "value": 4 } }, "87b7c9c1dbaa492f8b16883f0b86edea": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_6746fd5defb744b7a2b942e5ac957881", "IPY_MODEL_86c7d46257bf44d3a8bf5084e8dc142a", "IPY_MODEL_38cf14d49e554edaa17109a726ff70b5" ], "layout": "IPY_MODEL_942db27ef44d4fc28cd27d8557b4c79f" } }, "8ff495a1bd45497dbd7cdbd2c36e2080": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "942db27ef44d4fc28cd27d8557b4c79f": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "958c7ee6595f4dfbbab8d4366d4fbfd5": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "a0ef3882f90e4c87987846ba5d40878c": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "b851d6883c9f466ebe94d70bf9f9f8d6": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "d4df9e677d6d47f88dcc88680753f87f": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "db29d3f772b54102932d4264cbed0f67": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "dd77821774f64a42addbd009fc56e952": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "e6e06336c53c442e9a19466d3e959ef7": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_3e6a83f852b34d099ebee7ce2b994e80", "IPY_MODEL_0c41ca80d0834a129f9ee68f5e6f46c5", "IPY_MODEL_18d2f36a79cb464493c4bb210c7f2d3d" ], "layout": "IPY_MODEL_a0ef3882f90e4c87987846ba5d40878c" } }, "f3f1103dc3f34671833adbe0b2dc07f4": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "f889c274f45f40f39d285ffdf81d8927": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "f899c73fab574bde9f0e06c68d2c6f65": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }